WIP: add metrics for workloads count #4350
Conversation
k8s-ci-robot
added
kind/feature
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kannon92 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
size/L
✅ Deploy Preview for kubernetes-sigs-kueue canceled.
|
pkg/metrics/metrics.go
Outdated
| // Metrics counting usage of frameworks | ||
|
|
||
| // Counter of jobs that are managed by Kueue | ||
| JobWorkloadCounter = prometheus.NewCounterVec( |
There was a problem hiding this comment.
Please don't multiply the metrics by the framework name. Framework name should be a label in the metrics.
There was a problem hiding this comment.
Basically, increasing kind of metrics affect scheduling performance since the metrics counter need to take a lock every time.
So, I would recommend adding the kind of workload type like batch/v1 Job to the existing Workload counting metrics as @mimowo mentioned.
There was a problem hiding this comment.
Do you all think I should have two metrics.
one for supported frameworks and the other for external?
I wasn’t sure if exposing the GVK for external frameworks would be valid.
There was a problem hiding this comment.
I like the idea of extending the existing workload metrics more than entirely new ones. If this is risky we may use a feature gate for a release or two
There was a problem hiding this comment.
you can aggregate the metrics at the level of grafana, by summing over the CQ names. Then you will display results by frameworkName
There was a problem hiding this comment.
Are you okay with including external frameworks in that same label?
There was a problem hiding this comment.
I'm ok with that, every workload needs to belong to some framework, either built-in or external, so otherwise we would need to say "". I guess user needs to have some other way (if she / he wants) to determine which are external and which built-in.
There was a problem hiding this comment.
Basically, increasing kind of metrics affect scheduling performance since the metrics counter need to take a lock every time
Prometheus metric counter is thread safe no?
There was a problem hiding this comment.
Prometheus metric counter is thread safe no?
Yes, that is thread safe. I meant metrics counter take a lock internally. We do not need to take a lock in our side.
k8s-ci-robot
added
the
do-not-merge/hold
k8s-ci-robot
changed the title
k8s-ci-robot
added
the
do-not-merge/work-in-progress
|
Leaving the first proper pass to @gabesaba, just raising #4350 (comment) as a top-level remark. |
k8s-ci-robot
added
size/M
|
@kannon92: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I'm trying to find the place to put the count of workloads. Anyone have an idea? |
|
/assign |
What type of PR is this?
/kind feature
What this PR does / why we need it:
We will provide an operator to manage the configuration of Kueue and we expect to have various customers with varrying frameworks supported for Kueue.
Over time we know we would like to see what kind of Kinds are managed by Kueue and how many.
We want to propose a counter per in-tree framework that counts the number of workloads kueue managed and whether or not the workload is actually enabled.
Which issue(s) this PR fixes:
Fixes: #4336
Special notes for your reviewer:
This is open for comments.
Does this PR introduce a user-facing change?